Augmented reality for sensor applications

ABSTRACT

System, method, and media for an augmented reality interface for sensor applications. Machines making up a particular production or processing facility are instrumented with one or more sensors for monitoring their operation and status and labeled with machine-readable tags. When viewed by a technician through an augmented reality display, the machine-readable tags can be recognized using a computer-vision system and the associated machines can then be annotated with the relevant sensor and diagnostic data. The sensors may further form a mesh network in communication with a head-mounted display for an augmented reality system, eliminating the needs for centralized networking connections.

BACKGROUND 1. Field

Embodiments of the invention generally relate to augmented reality and,more particularly, to the use of machine-readable indicia in combinationwith sensors to present sensor data in an augmented reality display.

2. Related Art

Traditionally, processing plants and machinery are managed from acentral control unit such as a SCADA supervisory control system. Suchsystems use a graphical user interface on the central controller tomanage controller set point changes, field sensor data acquisition, andactuators. However, when a technician needs physical access to themachinery, they lose access to the dynamic display and management ofdata. Furthermore, the schematic display of data, while useful forhigh-level process control, can make it difficult to identify aparticular physical machine in need of servicing. As such, there is aneed for a display that can integrate sensor and diagnostic data forprocess machinery with real-world scene data of the machinery to allow atechnician to quickly identify which piece of physical machinery isassociated with an aberrant sensor reading.

SUMMARY

Embodiments of the invention address the above-described need byproviding for an augmented reality interface for sensor applications. Inparticular, in a first embodiment, the invention includes a system forproviding an augmented reality interface for sensor applications,comprising a plurality of sensors, wherein each sensor comprises atransducer and a communication interface, is configured to gather sensordata for a machine, and has an associated machine-readable indicium; acamera, configured to capture imagery of a scene including amachine-readable indicium associated with a sensor of the plurality ofsensors, a processor, programmed to receive the camera imagery andrecognize the machine-readable indicium, receive sensor data from the atleast one sensor of the plurality of sensors, process the sensor data togenerate a data visualization associated with the sensor of theplurality of sensors, generate an augmented display including the sceneand the data visualization associated with the sensor of the pluralityof sensors, and a portable display, configured to display the augmentedscene in proximity to the scene.

In a second embodiment, the invention includes a method for generatingan augmented reality display of a scene, comprising the steps ofreceiving, from a camera, imagery of a scene, wherein the imagery of thescene includes a machine with at least one machine-readable indiciumassociated with a sensor, processing the imagery of the scene toidentify the machine-readable indicium, determining a sensor associatedwith the machine-readable indicium, retrieving, from the sensor, sensordata for the machine, generating a data visualization based on thesensor data for the machine, overlaying the data visualization on theimagery of the scene in proximity to the machine-readable indicium toform an augmented scene, and displaying the augmented scene on a displayin proximity to the scene.

In a third embodiment, the invention includes one or morecomputer-storage media storing computer executable instructions that,when executed by a processor, perform a method of generating anaugmented reality display of a scene, the method comprising the steps ofreceiving data from a plurality of sensors associated with a respectiveplurality of machines, wherein each sensor of the plurality of sensorsis further associated with a respective machine-readable indicium,receiving imagery of a scene, wherein the imagery includes at least onevisible machine-readable indicium, determining, based on the visiblemachine-readable indicium, a sensor of the plurality of sensors,generating a data visualization from the data received from thedetermined sensor, overlaying the data visualization on the imagery ofthe scene to form an augmented scene, and displaying the augmented sceneon a display to the user.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Other aspectsand advantages of the current invention will be apparent from thefollowing detailed description of the embodiments and the accompanyingdrawing figures.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

Embodiments of the invention are described in detail below withreference to the attached drawing figures, wherein:

FIG. 1 depicts an exemplary hardware platform for certain embodiments ofthe invention;

FIG. 2 depicts a diagram depicting an exemplary operational environmentfor embodiments of the invention;

FIG. 3 depicts an exemplary scene the might be displayed in accordancewith embodiments of the invention; and

FIG. 4 depicts a flowchart illustrating the operation of a method inaccordance with embodiments of the invention.

The drawing figures do not limit the invention to the specificembodiments disclosed and described herein. The drawings are notnecessarily to scale, emphasis instead being placed upon clearlyillustrating the principles of the invention.

DETAILED DESCRIPTION

At a high level, embodiments of the invention provide for an augmentedreality interface for sensor applications. Machines making up aparticular production or processing facility are instrumented withsensors for monitoring their operation and labeled with machine-readabletags. When viewed through an augmented reality display, themachine-readable tags can be recognized using a computer-vision systemand the associated machines can then be annotated with the relevantsensor and diagnostic data.

The subject matter of embodiments of the invention is described indetail below to meet statutory requirements; however, the descriptionitself is not intended to limit the scope of claims. Rather, the claimedsubject matter might be embodied in other ways to include differentsteps or combinations of steps similar to the ones described in thisdocument, in conjunction with other present or future technologies.Minor variations from the description below will be obvious to oneskilled in the art, and are intended to be captured within the scope ofthe claimed invention. Terms should not be interpreted as implying anyparticular ordering of various steps described unless the order ofindividual steps is explicitly described.

The following detailed description of embodiments of the inventionreferences the accompanying drawings that illustrate specificembodiments in which the invention can be practiced. The embodiments areintended to describe aspects of the invention in sufficient detail toenable those skilled in the art to practice the invention. Otherembodiments can be utilized and changes can be made without departingfrom the scope of the invention. The following detailed description is,therefore, not to be taken in a limiting sense. The scope of embodimentsof the invention is defined only by the appended claims, along with thefull scope of equivalents to which such claims are entitled.

In this description, references to “one embodiment,” “an embodiment,” or“embodiments” mean that the feature or features being referred to areincluded in at least one embodiment of the technology. Separatereference to “one embodiment” “an embodiment”, or “embodiments” in thisdescription do not necessarily refer to the same embodiment and are alsonot mutually exclusive unless so stated and/or except as will be readilyapparent to those skilled in the art from the description. For example,a feature, structure, or act described in one embodiment may also beincluded in other embodiments, but is not necessarily included. Thus,the technology can include a variety of combinations and/or integrationsof the embodiments described herein.

Turning first to FIG. 1, an exemplary hardware platform for certainembodiments of the invention is depicted. Computer 102 can be a desktopcomputer, a laptop computer, a server computer, a mobile device such asa smartphone or tablet, or any other form factor of general- orspecial-purpose computing device. Depicted with computer 102 are severalcomponents, for illustrative purposes. In some embodiments, certaincomponents may be arranged differently or absent. Additional componentsmay also be present. Included in computer 102 is system bus 104, wherebyother components of computer 102 can communicate with each other. Incertain embodiments, there may be multiple busses or components maycommunicate with each other directly. Connected to system bus 104 iscentral processing unit (CPU) 106. Also attached to system bus 104 areone or more random-access memory (RAM) modules. Also attached to systembus 104 is graphics card 110. In some embodiments, graphics card 104 maynot be a physically separate card, but rather may be integrated into themotherboard or the CPU 106. In some embodiments, graphics card 110 has aseparate graphics-processing unit (GPU) 112, which can be used forgraphics processing or for general purpose computing (GPGPU). Also ongraphics card 110 is GPU memory 114. Connected (directly or indirectly)to graphics card 110 is display 116 for user interaction. In someembodiments no display is present, while in others it is integrated intocomputer 102. Similarly, peripherals such as keyboard 118 and mouse 120are connected to system bus 104. Like display 116, these peripherals maybe integrated into computer 102 or absent. Also connected to system bus104 is local storage 122, which may be any form of computer-readablemedia, and may be internally installed in computer 102 or externally andremoveably attached.

Computer-readable media include both volatile and nonvolatile media,removable and nonremovable media, and contemplate media readable by adatabase. For example, computer-readable media include (but are notlimited to) RAM, ROM, EEPROM, flash memory or other memory technology,CD-ROM, digital versatile discs (DVD), holographic media or otheroptical disc storage, magnetic cassettes, magnetic tape, magnetic diskstorage, and other magnetic storage devices. These technologies canstore data temporarily or permanently. However, unless explicitlyspecified otherwise, the term “computer-readable media” should not beconstrued to include physical, but transitory, forms of signaltransmission such as radio broadcasts, electrical signals through awire, or light pulses through a fiber-optic cable. Examples of storedinformation include computer-usable instructions, data structures,program modules, and other data representations.

Finally, network interface card (NIC) 124 is also attached to system bus104 and allows computer 102 to communicate over a network such asnetwork 126. NIC 124 can be any form of network interface known in theart, such as Ethernet, ATM, fiber, Bluetooth, or WiFi (i.e., the IEEE802.11 family of standards). NIC 124 connects computer 102 to localnetwork 126, which may also include one or more other computers, such ascomputer 128, and network storage, such as data store 130. Generally, adata store such as data store 130 may be any repository from whichinformation can be stored and retrieved as needed. Examples of datastores include relational or object oriented databases, spreadsheets,file systems, flat files, directory services such as LDAP and ActiveDirectory, or email storage systems. A data store may be accessible viaa complex API (such as, for example, Structured Query Language), asimple API providing only read, write and seek operations, or any levelof complexity in between. Some data stores may additionally providemanagement functions for data sets stored therein such as backup orversioning. Data stores can be local to a single computer such ascomputer 128, accessible on a local network such as local network 126,or remotely accessible over Internet 132. Local network 126 is in turnconnected to Internet 132, which connects many networks such as localnetwork 126, remote network 134 or directly attached computers such ascomputer 136. In some embodiments, computer 102 can itself be directlyconnected to Internet 132.

Turning now to FIG. 2, a diagram depicting an exemplary operationalenvironment for embodiments of the invention is depicted. In thedepicted environment, user 202 is a technician supporting one or moreitems of machinery, including machine 204, machine 206, and machine 208.The term “machine” (as used in reference to machines such as machine204, 206, or 208) should be construed broadly as any object thatperforms a task and may require maintenance or servicing. As depicted,machine 204 is a curing oven, machine 206 is a part picker, and machine208 is a printer. However, a machine could be any other component of anassembly line, such as an individual machine tool, or a completeassembly line. Other machines are also contemplated. For example, amachine could be an individual server, storage rack, or networkingelement in a data center. Alternatively, a machine could be anautomobile, or a particular component of an automobile. Machines 204,206 and 208 could be identical (for example, a set of batch mixers),related (for example, part of the same production process), orcompletely unrelated.

In some embodiments, equipment including machines 204, 206 and 208 iscontrolled by a central controller 210 (for example, a SCADA system). Inother embodiments, the various other components described hereincommunicate directly with each other in a peer-to-peer or mesh networkfashion and do not require a central controller. Central controller 210may monitor, control, track and/or log data from the equipment. Forexample, where the equipment makes up a production line, centralcontroller 210 may guide partially competed product from machine 204 tomachine 206 to machine 208. Central controller may also monitor theequipment for faults or other problems, as discussed in greater detailbelow. Central controller 210 may communicate with the controlledequipment via a wired network or wireless network including wirelessaccess point 212, as described above with respect to FIG. 1. Inparticular, central controller may monitor the outputs of sensors 214(corresponding to machine 204), sensors 216 (corresponding to machine206) and sensors 218 (corresponding to machine 208).

Sensors such as sensors 214, 216, and 218 include some sort oftransducer and a communication interface and may provide a variety ofdata regarding the operation of their respective machines. For example,a sensor may monitor the temperature of a particular production step, avibrational mode of a machine, pressure in an enclosed vessel, rotationof a component (via an optical transducer or a Hall effect sensor), pHof a mixture, voltage or current across a particular electric orelectronic component, imagery of a process or component in a visual,infrared or ultraviolet spectrum, or any other physical phenomenaassociated with the machine. As depicted with respect to machines 204and 208, a single machine may have multiple associated sensors of thesame type or disparate types. Sensors may be integrated into theequipment or retrofit.

In some embodiments, sensors 214, 216, and 218 communicate with centralcontroller 210. Where the sensors are integrated into the associatedmachine, the sensors may communicate via the network interface of themachine. Alternatively, or where the sensors are retrofit on themachines, the sensors may communicate wirelessly with central controller210, head-mounted display 220, respective machine-readable indicia 224,226, and 228, and/or the other components of the system. Based on thedata collected by the sensors, the system may determine that a machineis in an aberrant condition or otherwise experiencing a fault. Forexample, central controller 210 may process the data from sensors 218 todetermine that a fault has occurred with machine 208, and that servicingis required. However, particularly where a large number of identicalmachines are present, it may be difficult for technician 202 to identifywhich machine needs servicing, and (if multiple machines need servicing)which fault is associated with which machine.

In some embodiments, data from sensors 214, 216, and 218 may beprocessed at central controller 210 using machine learning techniques todetermine an operational state for the machine. For example, the systemmay process the sensor data during normal operation to learn the rangeof values from the sensors when the machine is operating correctly. Ifthe sensors later return values outside of the normal range, it mayindicate that a fault is imminent or has occurred. Thus, in addition todetecting that faults have occurred and repair is needed, the sensorscan allow for “predictive maintenance” to be performed before a faultactually occurs based on changing sensors values. For example, if amachine has an associated vibration sensor, then the system may recordwhat vibrational modes are present and their intensity under normaloperating conditions. If new vibrational modes are subsequentlydetected, this may indicate that a bearing is worn and should bereplaced before it fails completely and causes additional damage to themachine.

Accordingly, technician 202 may be equipped with a head-mounted display(HMD) 220. In some embodiments, head-mounted display may be an opticalHMD (also known as an optical see-through HMD) which overlays projectedimagery on a partially transparent lens. In other embodiments, HMD 220may be a digital HMD (also known as a video see-through HMD) whichcaptures digital imagery using one or more cameras, overlays theprojected imagery on the captured imagery, and displays the combinedimages on one or more displays (such as LCD or OLED displays) positionedin front of the eyes of the wearer. Because the wearer sees both theiractual surroundings and the overlain information, this is also known as“augmented reality.” Broadly, HMD 220 can be any display allowingtechnician 202 to see a view of the real world surroundings (whetherdirectly or via a video camera/video display system) in combination withoverlain information, in proximity to the real-world surroundings beingdisplayed. For example, HMD 220 could be a smartphone configured tocapture imagery from a camera on one side and display augmented imageryon the display on the opposite side.

In some embodiments, HMD 220 includes a camera (or cameras) to captureinfrared imagery. This can be used to collect temperature informationfor the machinery, to recognize information displayed in the infraredspectrum (as discussed below), or for any other purpose. Thus, forexample, HMD 220 could include two visible-light cameras and oneinfrared camera, two cameras sensitive in both the visible and infraredspectra, or a single infrared camera in conjunction with a see-throughHMD.

In order to correctly position the overlain information, machines 204,206, and 208 may also include machine-readable indicia 224, 226, and228. Broadly, machine-readable indicia include any marker encodinginformation that can be recognized by a computer-vision system. Forexample, the machines may be labeled with bar codes, quick-recognition(QR) codes, or augmented reality tags. These machine-readable indiciaencode information usable by the system to aid technician 202 inservicing the associated machine.

For example, a bar code may encode only an 11-digit number, which canencode a serial number or other unique identifier for the machine, butnot encode detailed diagnostic information. An augmented reality tagallows for more robust recognition, but can typically encode fewersymbols. A QR code, by contrast, can encode more data (for example,specific values received from one or more associated sensors), butrequires higher-resolution imagery to successfully decode. One of skillin the art will appreciate that any of these, alone or in combination,can be employed to provide the system with information about the variousmachines. Broadly speaking, any type of visual machine-readable indicia(now known or later developed) is contemplated as being with the scopeof the invention. In some embodiments, multiple machine-readable indiciaare positioned on various surfaces of the associated machine so that atleast one indicium is visible regardless of the angle from whichtechnician 202 is viewing the machine.

Machine-readable indicia 224, 226, and 228 may further be active orpassive. As used in this specification, a passive machine-readableindicium is one that permanently displays a single encoded value. Forexample, a printed bar codes or augmented reality tags are examples ofpassive machine-readable indicia. By contrast, an activemachine-readable indicium can display a plurality of encoded values. Forexample, an electrophoretic (such as e-ink) can be used to display anarbitrary image such as a machine-readable indicium. Alternatively, astatic printed image can be used in combination with a backlight todisplay the machine-readable indicium only when the backlight is lit.Other types of display (such as a liquid-crystal display orlight-emitting diode display) can also be used to display amachine-readable indicium. Either active or passive machine-readableindicia can be displayed in the visible or infrared spectrum. Forexample, passive machine-readable indicia can use inks that contrastwhen illuminated in the selected spectrum. Similarly, activemachine-readable indicia can use light-emitting diodes that radiatelight in the selected spectrum.

As disclosed above, active machine-readable indicia can encode aplurality of machine-readable values. In particular, this can be usefulwhen a sensor for the associated machine detects a fault. In such acircumstance, the machine-readable indicia can activate (from displayingnothing to displaying a value) or change from displaying an “operatingcorrectly” value to displaying a “fault condition” value. Alternatively(or in addition), active machine-readable indicia can continuouslyupdate to display data received from one or more sensors for theassociated machine.

In some embodiments, active machine-readable indicia communicatedirectly with one or more associated sensors. Communication may be via awireless protocol such as Bluetooth®, Zigbee (i.e., the IEEE 802.15.4family of protocols), or WiFi. Alternatively, the machine-readableindicia may communicate with central controller 210, head-mounteddisplay 220, other machine-readable indicia, and/or any other componentsof the system. For example, the machine-readable indicia may form a meshnetwork (alone or in combination with other system components) tocommunicate with head-mounted display 220. In some embodiments,machine-readable indicia such as machine-readable indicium 216 may beintegrated into sensors such as sensor 226, allowing the sensor todisplay the encoded information directly on an exterior housing of thesensor. In other embodiments, both sensors and machine-readable indiciamay be integrated into a machine and communicate via a wired control buswith a controller for the machine.

Turning now to FIG. 3, an exemplary scene the might be displayed on HMD220 is depicted and referred to generally by reference numeral 300. Asdepicted, a user such as technician 202 is viewing an area with fourmachines: pressure vessel 302, pressure vessel 304, pressure vessel 306,and pump 308. As discussed above with respect to FIG. 2, pressure vessel302 has associated machine-readable indicia 312, pressure vessel 304 hasassociated machine-readable indicia 314, pressure vessel 306 hasassociated machine-readable indicia 316, and pump 308 has associatedmachine-readable indicia 318. Note that the base image includes both thecomponents that are part of the production system (such as the pressurevessels) as well as things that are not part of the production system(such as road 310 and the tree).

In addition to the real-world scene that would be visible to the unaidedvision of a user such as technician 202, scene 300 includes a number ofoverlays providing the user with information regarding the status andoperation of the system as a whole as well as the individual machinesmaking it up. In some embodiments, the status displays for a particularmachine are displayed in proximity to that machine. Status displays suchas status 322 may have a visual tether or tail tied to the associatedmachine-readable indicia to allow technician 202 to more easily locatethe associated machine. For example, the status 322 for machine 302shows that machine 302 is operating at 80% of capacity, and may need tobe serviced soon. Status 322 is associated with machine-readableindicium 312 so that, as technician 202 proceeds along road 310, status322 remains unobstructed and oriented so as to face technician 202.Similarly, status 324, associated with machine 304, shows that machine304 is operating within all design parameters. In some embodiments,machines with no warnings or fault conditions do not have associatedstatus displays so as to avoid cluttering display 300 with unneededinformation.

Similarly to machine 304, machine 308 has status display 328 indicatingthat it is functioning within normal parameters. In some embodiments,displays may be standardized across all machine types. In otherembodiments, each type of machine has its own status display to provideinformation particular to that machine. In still other embodiments, eachtype of machine has its own type of display, but displays arecolor-coded to indicate an alarm level. For example, displays 324 and328 may be colored green to indicate that there is no need for concern,while display 322 may be colored yellow to indicate a warning anddisplay 326 may be colored red to indicate an urgent alarm. As depictedfor status 322 and status 326, supplementary information for the warningor alarm may also be depicted.

Global information unrelated to a particular machine (or combininginformation for multiple machines) may also be included in display 300.For example, non-contextual displays 330 provide an overlay givingtechnician 202 access to the current date and time as well as area andlog-on information. In some embodiments, these non-contextual displays330 are positioned at fixed positions at the perimeter of display 300,so as to avoid obstructing vision. In some embodiments, supplementarydisplays 332 may also be included in display 300. For example, asdepicted, supplementary display 332 shows operating capacity for allpressure vessels together with thresholds for warning and urgent alarmstatuses. In some such embodiments, the user can configure thesupplementary displays to show the relevant information for the currentarea or task.

Turning now to FIG. 4, a flowchart illustrating the operation of amethod in accordance with embodiments of the invention is depicted andreferred to generally by reference numeral 400. The method begins at astep 402 where raw imagery from one or more cameras on HMD 220 fortechnician 202 is retrieved for processing. In some embodiments wheremultiple cameras are present, imagery from only a single camera isretrieved for processing. In some embodiments, camera imagery isprocessed on a processor of HMD 220. In other embodiments, HMD 220transmits the imagery from the camera(s) to another computer (such as,for example, central controller 210) for processing.

Next, at a step 404, the imagery is retrieved to identify themachine-readable indicia present in the image. Broadly speaking, theprecise techniques used to identify a machine-readable indicium willdepend on the type of indicium used. For example, techniques forrecognizing bar codes will differ from techniques for recognizing QRcodes, which will in turn differ from techniques for recognizingaugmented reality tags. One of skill in the art will appreciate thateach type of machine-readable indicia has an associated recognitionalgorithm. The steps described below are representative and should notbe considered limiting.

Broadly speaking, a recognition algorithm will start by converting theimagery received from the camera(s) to a single channel representingintensity (e.g., a grayscale image). In some embodiments, such as thoseutilizing infrared cameras to recognize machine-readable indicia, theimagery may include only a single channel that can be interpreted as theneeded intensity channel. Alternatively, a color image can be convertedif necessary to a desired representation (for example, to ahue/saturation/intensity representation) and the relevant channel (here,the intensity channel) extracted.

Once the relevant channel has been extracted, thresholding can be usedto determine connected components of the image with a gray level lessthan the threshold. In this way, the light-colored regions delimitingthe machine-readable indicia can be identified. In other embodiments,the connected components of the image with a gray level above thethreshold are instead determined to identify dark regions. In stillother embodiments, both light-colored and dark-colored regions areidentified with different thresholds. Because machine-readable indiciaare typically constructed using high-contrast colors, using dualthresholding may more accurately identify regions of the indicia.

Next, once the thresholded regions are identified, the external bordercontours are extracted from the identified regions. This allows thehigh-contrast shapes in the image to be identified. At this stage, theidentified shapes include any machine-readable indicia, but also anyother shapes with high-contrast edges present in the image. For example,letterforms, signs and windows present in the image may also beidentified as high-contrast shapes.

Once the high-contrast shapes have been identified, they are filtered toeliminate non-quadrilateral shapes. For example, the high-contrastletterforms mentioned above are eliminated from the recognized shapesbecause they are not quadrilaterals. In order to recognizequadrilaterals, the boundary of each shape can first be linearized(points lying in a narrow rectangular region are converted to a line).Shapes comprising up more than four such linear regions, or whose linearregions have endpoints that do not meet the endpoints of another linearregion, are non-quadrilaterals and can be eliminated.

For the remaining quadrilaterals, the points of intersection for thefour segments (i.e., the vertices of the quadrilateral) are extracted.The vertices of each extracted quadrilateral can then be used todetermine the unique projective (collinear) transformation for thatquadrilateral that will map it to the unit square. Applying thistransformation to the image allows the identified quadrilateral to becompared (in standardized form) to the reference form of the particularmachine-readable indicia being used. Then, a most likely encoded valuefor the particular machine-readable indicium and a confidence value thatthe identified region actually corresponds to the identified encodedvalue can be determined. If the confidence value is above apredetermined threshold, then the identified marker is associated withthe corresponding region in the original image. Where multiple imagesare to be processed (e.g., one image for each camera), the images may beprocessed separately and the confidence values for the correspondingregions of each image aggregated to determine whether a machine-readableindicium is present in that spatial region.

Once the set of machine-readable indicia present in the scene has beenidentified, processing can proceed to step 406, where data from thesensors associated with each identified machine-readable indicium isretrieved and processed. In some embodiments, the data is retrieved byquerying the sensors directly. In other embodiments, all sensorscommunicate their data to a central controller and the data from therelevant sensors is retrieved from the central controller. In stillother embodiments, the sensor data is retrieved from the machineassociated with the sensor.

In some embodiments, the raw data from the sensor is presented to theuser. For example, if a sensor detects temperature, then the sensor mayreport the temperature to the central controller, where it is forwardedto the HMD and presented directly to the user. In other embodiments, orfor other sensors, the data is processed before being presented to theuser. For example, if the sensor is a vibration sensor, the rawvibration waveform may not be informative. Instead, the sensor regularlyforwards the raw data to the server, which accumulates it and processesit using a Fast Fourier Transform to identify the vibrational modes.When the data for that sensor is requested by the HMD, the transformeddata is forwarded for presentation to the user, allowing the user toimmediately see any potentially problematic vibrational modes.

Once the data has been retrieved and processed, a display for thatsensor can be generated at step 408. Broadly speaking, any type of datavisualization can be employed to convey the relevant information to theuser. For example, time-series graphs of data may be presented for onetype of sensor, spectral data may be presented for a second type ofsensor, and a single instantaneous value can be presented for a thirdtype of sensor. As described above, data displays may be color coded(for example, by coloring their background) to represent a determinedstatus for the sensor or for the associated machine. In someembodiments, users may be able to cycle between differentrepresentations of the data.

Processing then proceeds to step 410 where the generated display foreach recognized machine-readable indicium is overlain on the display inproximity to the associated indicium or sensor. In some embodiments, asdescribed above, generated displays are connected to the image of theassociated machine-readable indicium via a marker. In other embodiments,the generated display is overlain directly on the associatedmachine-readable indicium. As discussed above, generated displays forsensors reporting no fault may be suppressed or suppressed only when oneor more other sensors indicate an aberrant condition. Where the systemincludes a display for each eye, the generated displays may be overlainon a single eye to allow transparency or on both eyes to provide athree-dimensional effect.

Many different arrangements of the various components depicted, as wellas components not shown, are possible without departing from the scopeof the claims below. Embodiments of the invention have been describedwith the intent to be illustrative rather than restrictive. Alternativeembodiments will become apparent to readers of this disclosure after andbecause of reading it. Alternative means of implementing theaforementioned can be completed without departing from the scope of theclaims below. Certain features and subcombinations are of utility andmay be employed without reference to other features and subcombinationsand are contemplated within the scope of the claims. Although theinvention has been described with reference to the embodimentsillustrated in the attached drawing figures, it is noted thatequivalents may be employed and substitutions made herein withoutdeparting from the scope of the invention as recited in the claims.

Having thus described various embodiments of the invention, what isclaimed as new and desired to be protected by Letters Patent includesthe following:
 1. A system for providing an augmented reality interfacefor sensor applications, comprising: a plurality of sensors, whereineach sensor: comprises a transducer and a communication interface; isconfigured to gather raw sensor data for a machine, wherein the rawsensor data is not suitable for visualization; and has an associatedmachine-readable indicium; a camera, configured to capture imagery of ascene including a machine-readable indicium associated with a sensor ofthe plurality of sensors; an infrared camera generating infrared imageryof the machine; a central controller programmed to: receive the rawsensor data from the plurality of sensors, and process the raw sensordata into sensor data suitable for visualization, a processor,programmed to: receive the camera imagery and recognize themachine-readable indicium, determine the sensor of the plurality ofsensors associated with the machine-readable indicium; upondetermination of the sensor, request, by a head-mounted display, thesensor data suitable for visualization; receive the sensor data suitablefor visualization from the central controller; process the sensor datasuitable for visualization to generate a data visualization associatedwith the sensor of the plurality of sensors; determine an operatingcapacity of the machine associated with the sensor based on the rawsensor data; determine the temperature of the machine based at least inpart on the infrared imagery; generate an augmented display includingthe scene, the data visualization associated with the sensor of theplurality of sensors, and the temperature of the machine determined fromthe infrared imagery; present the data visualization on a datavisualization display, wherein a type of the data visualization displayis indicative of a type of machine associated with the sensor, whereinthe data visualization display presents an option for the user to cyclethrough various types of data representations, wherein the datavisualization includes the operating capacity relative to operationalthresholds for alarm statuses of the machine and the temperature of themachine determined from the infrared reading; and suppress the datavisualization associated with the sensor of the plurality of sensorsonly when one or more other sensors of the plurality of sensors indicatean aberrant condition; and a portable display, configured to display theaugmented scene in proximity to the scene, wherein a non-contextualdisplay presents non-contextual information, and is configured to befixed, by a user, at a location on the augmented display to avoidobstructing the augmented scene.
 2. The system of claim 1, wherein thecamera, the display, and the processor are integrated into a singleunit.
 3. The system of claim 2, wherein the single unit is thehead-mounted display.
 4. The system of claim 1, wherein the camera andthe display are integrated into the head-mounted display, and whereinthe machine-readable indicium is displayed by electrophoresis.
 5. Thesystem of claim 1, wherein the machine-readable indicium is anartificial reality tag.
 6. The system of claim 1, wherein the datavisualization is color-coded to indicate a status of the machine.
 7. Thesystem of claim 1, wherein the machine-readable indicium includes aplurality of infrared light-emitting diodes.
 8. A method for generatingan augmented reality display of a scene, comprising the steps of:receiving, from a camera, imagery of a scene; receiving, from aninfrared camera, infrared imagery of the scene, wherein the imagery ofthe scene includes a machine with at least one machine-readable indiciumassociated with a sensor; processing the imagery of the scene toidentify the machine-readable indicium; processing the infrared imageryof the scene to determine a temperature; determining the sensorassociated with the machine-readable indicium; upon determination of thesensor, request, by a head-mounted display, raw sensor data for themachine associated with the machine-readable indicium; retrieving, by acentral controller, raw sensor data from the sensor, wherein the rawsensor data is not suitable to visualization; processing, by the centralcontroller, the raw sensor data into the sensor data suitable forvisualization; retrieving, from the central controller, the raw sensordata for the machine; determining an operating capacity of the machinebased on the raw sensor data; generating a data visualization based onthe sensor data suitable for visualization for the machine, presentingthe data visualization on a data visualization display, wherein a typeof the data visualization display is indicative of a machine type of themachine associated with the sensor, wherein the data visualizationdisplay presents an option for the user to cycle through various typesof data representations, wherein the data visualization includes theoperating capacity relative to operational thresholds for alarm statusesof the machine and the temperature determined from the infrared imagery;overlaying the data visualization on the imagery of the scene inproximity to the machine-readable indicium to form an augmented scenewhen the raw sensor data is indicative of an aberrant condition;suppressing data visualization associated with at least one other sensoron the augmented screen; displaying the augmented scene on a display inproximity to the scene; displaying at least one supplemental display onthe display including non-contextual information, wherein thesupplemental display is configurable by the user, and wherein thesupplemental display is fixed at a location by the user to avoidobstructing the augmented scene.
 9. The method of claim 8, wherein thecamera and the display are integrated into the head-mounted display. 10.The method of claim 8, wherein the machine-readable indicium is a barcode.
 11. The method of claim 8, wherein the data visualization is colorcoded to indicate a status of the machine.
 12. The method of claim 8,wherein the data visualization is suppressed from the augmented scene ifthe raw sensor data for the machine indicates that the machine isoperating normally.
 13. The method of claim 8, wherein data from aplurality of sensors for the machine is combined to generate the datavisualization.
 14. One or more non-transitory computer-storage mediastoring computer executable instructions that, when executed by aprocessor, perform a method of generating an augmented reality displayof a scene, the method comprising the steps of: receiving data from aplurality of sensors associated with a respective plurality of machines,wherein each sensor of the plurality of sensors is further associatedwith a respective machine-readable indicium; receiving imagery of ascene; receiving infrared imagery of the scene, wherein the imageryincludes at least one visible machine-readable indicium; determining,based on the visible machine-readable indicium, a sensor of theplurality of sensors; upon determination of the sensor, request, by ahead-mounted display, raw sensor data for the machine associated withthe machine-readable indicium; receiving, by a central controller, rawsensor data from the sensor, wherein the raw sensor data is not suitablefor visualization; processing the raw sensor data into the sensor datasuitable for visualization; determining an operating capacity of therespective machine based on the raw sensor data received from thesensor; determining a temperature of the respective machine based atleast on part on the infrared imagery; generating a data visualizationfrom the sensor data suitable for visualization received from thedetermined sensor by the central controller, present the datavisualization on a data visualization display, wherein a type of thedata visualization display is indicative of a type of machine associatedwith the determined sensor, wherein the data visualization displaypresents an option for the user to cycle through various types of datarepresentations, wherein the data visualization includes the operatingcapacity relative to operational thresholds for alarm statuses of themachine and the temperature determined from the infrared imagery;displaying the data visualization on a portion of a display separatefrom the imagery of the scene to form an augmented scene; suppressingthe data visualization when one or more other sensors indicate anaberrant condition, displaying data visualization associated with theone or more other sensors indicative of the aberrant condition;displaying the augmented scene on the display to the user; displayingnon-contextual information by a non-contextual display wherein thenon-contextual display is configurable by the user, and wherein a fixedlocation of the non-contextual display is configurable by the user toavoid obstructing the augmented scene.
 15. The media of claim 14,wherein the display and the camera are integrated into the head-mounteddisplay.
 16. The media of claim 14, wherein the display and the cameraare integrated into a smartphone.
 17. The media of claim 14, wherein thescene includes a plurality of machine-readable indicia other than thevisible machine-readable indicia, and wherein data visualizations foreach of the associated plurality of sensors are overlain on the scene inthe augmented scene.
 18. The media of claim 14, wherein the datavisualization is color coded to indicate a status of the machine, andwherein the data visualization is displayed over only one eye of theuser.
 19. The media of claim 14, wherein the data visualization issuppressed from the augmented scene if the data received from thedetermined sensor indicates that the machine is operating normally. 20.The media of claim 14, wherein the machine-readable indicium is aquick-response code.